1 link tagged with all of: github + connection pooling + replica lag
Click any tag below to further narrow down your results
Links
GitHub's September 13 incident stemmed from a background data-cleanup job that overwhelmed a shared database cluster because the safeguard only monitored replica lag, not the primary's connection limit. Long request timeouts and retry loops made recovery harder and exposed how offline jobs can silently tank online systems.
- The safeguard watched the wrong metric (replica lag stayed healthy while the primary hit its connection limit), showing that health signals must track the actual bottleneck, not just a proxy.
- Long timeouts are invisible failures: they don't surface during normal operation like short timeouts do, but instead accumulate in-flight requests that starve the system of resources.
- Offline jobs against shared online databases need rate-limiting and load-shedding controls built in advance—you can't improvise this during an incident when you're already saturated.
- The fixes (rate-limits, better monitoring, request timeouts, database sharding) all add complexity, which is the eternal tradeoff of preventing incidents.